The Wolfram Data Science Platform: Data Science in the Cloud

MouseOver

MouseOver

Dillon Tracy

Senior Kernel Developer, Wolfram Research

Wolfram Cloud Platforms

Wolfram Programming Cloud: launched summer 2014

Optimized for programming projects, startups, prototyping

Other Platforms: Mathematica Online

Mathematica Online: launched September 2014

Extends the desktop Mathematica experience into the cloud

The Wolfram Data Science Platform

Emphasis on import-analyze-publish workflow

Why do data science in the cloud?  Accessibility and reach

Cloudifying Computable Documents

API functions democratize access to computation

Cloudifying computable documents democratizes access to results

Solves the “my boss doesn’t have Mathematica” problem

GraphicsGrid[
Partition[
Tooltip[Button[cdfThumb[#],NotebookOpen[FileNameJoin[{Directory[],"../WDS 2013/CDF-Examples",#}]],Appearance->Automatic,FrameMargins->10],#]&/@cdfSamples,
3,3,{1,1},{}
],ImageSize->600,Dividers->LightGray
]

Data Science Platform Workflows

The Wolfram Data Science Platform supports in-cloud, uploaded, url-based, and external API-based data sources.

A data source:

$url="https://docs.google.com/spreadsheet/pub?key=0An-lvWEXFqTvdElnLUNEOG5QTUJJaUh2aEpabUdIbEE&single=true&gid=0&output=csv";

SemanticImport attempts to recognize entities in structured data:

$CurrentData=SemanticImport[$url]

$CurrentData[All,{"date","elec (kWh)"}]//DateListPlot

Trees

An inventory of tree species and locations in Champaign, Illinois:

$src="ChampaignTrees.csv";

Take[Import[$src],4]//Column

d=Quiet@SemanticImport[FileNameJoin[{Directory[],$src}],{"Plant",Automatic,Automatic,Automatic,Automatic,Automatic},
"Rows"
];

Take[d,4]//Grid[#,Dividers->LightGray]&

LinguisticAssistant["Properties"]//Length

LinguisticAssistant["FoliageColor"]

Document Generators

Computable documents, notebook templating, and scheduled tasks converge in document generators.

A document generator consists of a template, and may also have a schedule and logic for computing template parameters.

Examine a template:

NotebookPut@Import["ExampleData/BasicTemplate.nb"]

Create a basic document generator from this template:

Quiet@RemoveScheduledTask[CloudObject["WTC/r1"]];

co=CloudDeploy[DocumentGenerator[File["ExampleData/BasicTemplate.nb"],None],"WTC/r1"]

Trigger the generator to run synchronously using EvaluateScheduledTask.  Evaluation is taking place in the cloud.

With[{o=co},EvaluateScheduledTask[o]]

Trigger the generator to run asynchronously using RunScheduledTask.  Evaluation is taking place in the cloud.

With[{o=co},RunScheduledTask[o]]

Document Generators: Utilities Again

Here is the previous utilities example, deployed from code:

Quiet@RemoveScheduledTask[CloudObject["WTC/Utilities-report"]];

co=CloudDeploy[DocumentGenerator[
File["Utilities-report.nb"],
"Monthly"
],CloudObject["WTC/Utilities-report"]
]

Trigger the generator to run asynchronously:

With[{o=co},RunScheduledTask[o]]

Document Generators: Stock Report

Create a document generator that produces a daily report on the performance over the past m days of the n trucking industry stocks with the highest trading volumes on the previous day. Here is the template:

NotebookPut@Import["ExampleData/StocksTemplate.nb"]

Deploy the generator with a suitable driver:

Clear[driver];
driver[m_Integer, n_Integer]:=Block[{members,reportOn},
members=FinancialData["Trucking","Members"];
reportOn=Take[Reverse[SortBy[
({#,Quiet[FinancialData[#,"Volume",{Yesterday}]]}&/@members)/.{{_,_Missing|_FinancialData}->Sequence[],{t_String,{{_List,i_Integer}}}:>{t,i}},
Last
]],n][[All,1]];
<|
"Start":>DateString[DatePlus[Today,-m],"DateShort"],
"Stocks"->(
<|"Name"->FinancialData[#,"Company"],"Ticker"->#|>&/@reportOn)
|>
]&;

With[{dest="WTC/trucking"},
Quiet@RemoveScheduledTask[CloudObject[dest]];
co=CloudDeploy[DocumentGenerator[
File["ExampleData/StocksTemplate.nb"],
driver[30,5],
DateObject[{_,_,Tuesday|Wednesday|Thursday|Friday|Saturday,1}]
],dest]
]

The generation time for this report is potentially long, so this is an example of a case where asynchronous evaluation is preferable:

With[{o=co},RunScheduledTask[o]]

Summary

DSP is designed around an import-analyze-publish workflow: data in, document out

Semantic interpretation enables knowledge-based analytics

Data science in the cloud solves the dissemination problem

Cloud-only and desktop hybrid workflows supported

dillont@wolfram.com

Team members also at the conference:

Deepak Balachandar
Brian Lee
Jeremy Lee
Adam Martinek
Clayton Voyles

Init

Init

Needs["JLink`"];

Needs["DatabaseLink`"];

Needs["HadoopLink`"];

Needs["CorporateAnalysisUtilities`"];

SetDirectory[NotebookDirectory[]]

PrependTo[$Path,"../WTC 2014"];

PrependTo[$Path,"../WDS 2013/CDF-Examples"];

PrependTo[$Path,"./cdf thumbs"];

cdfSamples=Complement[
Last[FileNameSplit[#]]&/@FileNames["*.cdf","../WDS 2013/CDF-Examples"],
{"CDF and ReportGeneration.cdf"}
]

Scan[
(cdfThumb[#]=Import[FileBaseName[#]<>".jpg"];&),
cdfSamples
]

Off[FrontEndObject::notavail]

Cloud connection

PacletDirectoryAdd["/Users/dillont/eclipse/workspace/CloudPlatformComponents/apache-tomcat-6.0.41/data/PID-617d2f9e-84de-43c0-b38d-2ac266e8574f/"]

FindFile["CloudObject`"]

<<CloudObject`;

$CloudDebug=True;
$CloudDebugLevel=3;

$CloudDebug=False;
$CloudDebugLevel=3;

$CloudBase="http://localhost/";

$CloudBase="http://www.wolframcloud.com/";

$CloudBase="http://www.devel.wolframcloud.com/";

$CloudBase="http://www.test.wolframcloud.com/";

$CloudBase="http://www.wolframcloud.com/";

CloudConnect["dillont@wolfram.com",FromCharacterCode[{75,85,46,111,103,111,112,97}],"RememberMe"->True]

CDF Thumbs

ClearAll[nb];
Scan[
(nb[#]=Import[#];&),
cdfSamples
]

Clear[makeNbThumb];
makeNbThumb[nb_Notebook,width_:175]:=Module[
{nbObj=NotebookPut[nb],res},
res=makeNbThumb[nbObj,width];
NotebookClose[nbObj];
res
];
makeNbThumb[nbObj_NotebookObject,width_:175]:=Module[
{rs,top},
rs=Rasterize[nbObj,"RasterSize"];
top=With[{res=Rasterize[nbObj]},
If[res===$Failed,
Return[$Failed],
ImageTake[res,1.294*rs[[1]]]
]
];
ImageResize[top,width]
]

ClearAll[cdfThumb]

Scan[
(If[!MatchQ[cdfThumb[#],_Image|$Failed],
cdfThumb[#]=makeNbThumb[nb[#]];
]&),
Take[cdfSamples,6]
]

no=NotebookPut[nb[#]]&@cdfSamples[[12]]

t=(cdfThumb[#]=makeNbThumb[no])&@cdfSamples[[12]]

CreateDirectory["cdf thumbs"]

Export["cdf thumbs/"<>FileBaseName[#]<>".jpg",cdfThumb[#]]&/@cdfSamples

Special Cells

Picture Cell

PicturePrint[expr_,opts:OptionsPattern[Cell]]:=CellPrint[TextCell[expr,"Picture",opts,ShowStringCharacters->False]]

Overview Mouseover

OverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"OverviewSection"],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},BoxFrame->2,FrameStyle->None,RoundingRadius->5,Background->m8red[1]],Row[{Framed[Style[header,"OverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->m8red[1]]," » ",Framed[Style[subs,"OverviewSection"],FrameMargins->{{15,15},{5,5}},FrameStyle->m8red[1],BoxFrame->2,RoundingRadius->5,Background->White]}]]]],"OverviewSection"]

(*MapThread[OverviewMouseover,Transpose[{{"Chapter One","Chapter One topics, …"},{"Chapter Two","Chapter Two topics, …"},{"Chapter Three","Chapter Three topics, …"},{"Chapter Four","Chapter Four topics, …"}}]]*)

Large Overview Mouseover

LargeOverviewMouseover[header_,subs_]:=CellPrint@Cell[BoxData[ToBoxes@Mouseover[Framed[Style[header,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5],

Evaluate@If[subs=="",Framed[Style[header,"LargeOverviewSection",White],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange],Row[{Framed[Style[header,"LargeOverviewSection",{FontFamily->"Helvetica",White}],FrameMargins->{{15,15},{5,5}},FrameStyle->None,RoundingRadius->5,Background->Orange]," » ",Framed[Style[subs,"LargeOverviewSection",FontFamily->"Helvetica"],FrameMargins->{{15,15},{5,5}},FrameStyle->Orange,RoundingRadius->5,Background->White]}]]]],"LargeOverviewSection"]

MapThread[LargeOverviewMouseover,Transpose[{
{"Deployment","Scheduling Tasks, Computable Documents, Document Generators"}
}]
];

DG removal

RemoveScheduledTask[CloudObject["WDS/r1"]]

RemoveScheduledTask[CloudObject["WDS/r2"]]

RemoveScheduledTask[CloudObject["WDS/r3"]]

RemoveScheduledTask[CloudObject["WDS/r4"]]

Scratch

t=Import["ChampaignTrees.csv"];

t=Import["ChampaignTrees-short.csv"];

t[[1]]

Export["ChampaignTrees-short.csv",Prepend[RandomSample[t[[2;;]],500],t[[1]]]
]

Directory[]

t=DocumentGenerators[];

DocumentGeneratorInformation/@t

MapThread[LargeOverviewMouseover,Transpose[{
{"The Cloud","Platforms, Motivation"},
{"Components","Computable Documents, Scheduled Tasks, Document Templating"},
{"Workflow","Utilities, Trees, Stocks ..."}
}]
];